Coordinating Access to Computation and Data in Distributed Systems

Distributed computing has become a complex ecosystem of protocols and services for managing computation and data. Distributed applications are becoming complex as well. Users, particularly in scientific fields, wish to deploy large numbers of applications with complex dependencies and a large appetite for both computation and data. How may such systems and applications be brought together? I propose that applications deployed in distributed systems should be represented by an agent. The role of the agent is to transform an application’s abstract operations into concrete operations on the varying resources in a distributed system. The agent must hide the unpleasant aspects of individual resources while coordinating their activity in a manner specialized to each application. I examine four open problems in the design of agents for distributed computing. First, I explore a variety of techniques for coupling a job to an agent, informed by the experience of porting to different systems and deploying with several applications. Coupling an agent to a job via the debugger is by far the most reliable and usable technique and has acceptable overhead for scientific applications. Second, I describe the problem of coupling an agent to a variety of distributed data systems. This is difficult because of the subtle semantic differences between existing data interfaces. These differences result in the notion of an escaping error, which represents a runtime incompatibility between interfaces. Third, I present the problem of coupling an agent to a computation manager such as a batch system. This requires a careful discussion of errors in a distributed system. I develop ii a theory of error propagation and present the notion of error scope, which is needed to guide the propagation of escaping errors. Finally, I explain how an agent may coordinate the consumption of computation and data resources on behalf of a job. As a case study, I present BAD-FS, a system that executes data intensive batch workloads on faulty distributed systems. I conclude with quantitative evidence underscoring the importance of failure handling in distributed systems.